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ABSTRACT: The article, "Distributed Revisiting: An Analytic for Retention of Coherent Science 
Learning" is an interesting study that operates at the intersection of learning theory and learning 
analytics. The authors observe that the relationship between learning theory and research in the 
learning analytics field is constrained by several problems: 1) differences between the context of 
the research and the context of the studies that yielded the underlying theory; and 2) the 
challenge of constructing metrics that make accurate inferences about psychological or group 
processes. These problems are discussed in relation to Svihla, Wester, and Linn's paper. 
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Evidence from the distributed practice literature suggests that people acquire knowledge more 
effectively when study is divided into multiple sessions, and spread over time, rather than delivered in a 
single, longer session (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006; Donovan & Radosevich, 1999; 
Janiszewski, Noel, & Sawyer, 2003). Svihla, Wester, and Linn (2015, this issue) draw heavily on this body 
of research in their examination of students' tendency to revisit previously studied material in WISE 
(Web-based Inquiry Science Environment). The authors discovered that a general propensity to revisit 
was not a significant predictor of student retention, but a tendency to revisit certain key pages was 
positively associated with delayed post-test scores. Their findings suggest two needs: 1) develop 
analytics designed to measure the revisiting of critical, central ideas; and 2) find ways for instructors and 
instructional designers to better foster the revisiting of important content. 

The distributed practice literature is extensive and relatively consistent in its findings. It is perhaps 
instructive to quickly review some of the terminology used in the field. A distributed practice learning 
design takes the following format: 

• Study/Practice Episode; 

• Interstudy Interval (a period of time separating study/practice episodes); 

• Study/Practice Episode; 

• Retention Interval (the period of time separating the final study episode from the final test); 

• Final Test. 

This is also sometimes referred to as "spaced" study (i.e., a curricular design in which the same material 
is reviewed on two or more distinct occasions). When an interstudy interval does not exist, the study is 
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said to be "massed" (i.e., all the study occurs at once). Research suggests that spaced study produces 
higher levels of retention than massed study when total study time is kept constant. In a meta-analysis 
of distributed practice experiments, Cepeda et al. observed that 

"there is no hint that massed presentation was preferable to spaced, whether retention interval 
was very short (less than 1 min) or very long (over 30 days). This suggests that there is always a 
large benefit when information is studied on two separate occasions instead of only once". 

(2006, p. 359) 

A related body of literature has examined how spaced presentations might be best designed. What are 
the optimal choices for the length of the interstudy interval and retention interval? The findings from 
this research are less clear. Some evidence suggests that it depends, in part, on the type of content 
being taught (Donovan & Radosevich, 1999). A meta-analysis by Cepeda et al. (2006) suggests the 
existence of a general trend: to maximize student retention, the interstudy interval should increase as 
the desired retention interval increases. 

In Svihla, Wester, and Linn's (2015, this issue) paper, distributed practice theory is used to rationalize 
the study of student "revisiting" in WISE units. While this would seem to be a reasonable application of 
theory, it is instructive to examine the differences between the context of the research and the context 
of the studies that yielded the underlying theory. Historically, the literature on distributed practice has 
been heavily shaped by controlled experimental studies that tightly specified the timing and duration of 
teacher-directed tasks aimed at improving student recall. This is strikingly different than the classrooms 
described by Svihla et al., where students are engaged in authentic, student-directed, guided inquiry. 
Not surprisingly, student activity patterns in WISE exhibited large variations in the number and duration 
of the study episodes, the timing of interstudy intervals (when they exist), and the length of the 
retention interval (varying from 4 to 40 days). The study yielded significant results, but with extremely 
low effect size, making it difficult to draw firm conclusions, and calling into question the applicability of 
distributed practice theory to this particular context. 

Contextual differences were not the only challenge facing the research. Another concern was the subtle 
but critical difference between the notion of "revisiting," as described by the authors, and the 
"Study/Practice Episodes" of the distributed practice literature. A study/practice episode necessarily 
involves a directed effort to learn of a body of material. This is somewhat different from "revisiting" on 
WISE, where students do not necessarily view material with that intent. In fact, when they revisit a page, 
we don't know with certainty that they are trying to learn anything at all. Revisiting would appear to be 
a necessary condition for spaced practice, but it's not a sufficient condition. While we can assume a 
relationship exists between revisiting and spaced practice, it is unclear how strong that relationship is, or 
whether counts of revisits can usefully serve as a proxy indicator of distributed learning. 

The preceding limitations illustrate some of the challenges that can arise when researchers work at the 
intersection of learning theory and learning analytics. A good deal of the learning theory that we use 
today has emerged out of experimental studies where control groups were used to isolate variables. 
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This bears little resemblance to much of today's research in the learning analytics field, in which data 
tends to be collected from naturalistic learning settings. To what extent do these contextual differences 
limit the useful applicability of learning theory to learning analytics research? Some may be tempted to 
question the relevance of controlled studies, given the unnatural, artificial nature of experimental 
designs. However, I suggest the problem is not entirely one of relevance, but is rather more accurately 
viewed as a fundamental limitation in our ability to measure certain key phenomena of interest. For 
example, in the WISE study, the researchers couldn't measure the incidence of distributed practice 
directly, so instead they relied on a proxy indicator: counts of "revisiting." They were further hampered 
by tremendous variability in student practices and behaviours, making it more challenging to identify 
associations between "revisiting" and student performance. In other words, their problem was not that 
the learning theory was not relevant — most would agree that distributed practice is preferable to 
massed practice in WISE classrooms. Rather, the challenge was one of finding a reasonable way to 
determine when distributed practice was, and wasn't, occurring. Many learning analytics researchers 
face these types of challenges. They are forced to rely on uncertain proxy indicators (e.g., "revisiting" as 
a measure of spaced learning, "time online" as a measure of time on task, "opening a webpage" as a 
measure of reading, "asynchronous discussion" as a measure of collaboration, etc.) that may only 
roughly approximate the phenomena of interest. 

The authors are clear about the limitations of their study and the tentative nature of the research. They 
present their results as an exploratory first step. Their findings raise new questions, and there are 
several areas that could benefit from further study: 

1. Further experimentation with sophisticated, content-sensitive revisiting analytics. 

Given the results from this research, it would be worthwhile to explore whether the findings can be 
replicated in other online environments and with other learners. As the authors point out, previous 
studies have tended to examine students' general propensity to revisit material. The success of the 
current study suggests that it may be more effective to focus on the revisiting of specific curricular 
items. This would necessarily entail the identification of ways in which students might benefit from 
distributed practice, and the subsequent design of metrics that carefully track student repeat 
engagement with the relevant materials. It is notable that the Svihla et al. found that additional visits to 
some content (i.e., the static curriculum step) produced decreases in delayed post-test scores. We need 
to explore more deeply why revisiting some kinds of materials can have a negative impact. A better 
understanding of this phenomenon may allow us to develop more powerful analytic tools and design 
learning environments that engage students more productively. 

2. Consideration of the limitations of revisiting analytics. 

One of the interesting findings from the research was the discovery that revisiting metrics — even ones 
that target the revisiting of specific content — are not necessarily strong predictors of learning. While 
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Svihla et al. found a statistically significant relationship between revisiting the dynamic visualization and 
delayed post-test scores, the effect size was small. Thus we need to proceed cautiously when using 
revisiting metrics for diagnostic purposes, at least for the time being. However, it is possible that 
revisiting scores are more useful when presented in combination with other data. This is an area of 
research deserving of further exploration. One simple technique employed by my team on the Pepper 
project involves the production of diagnostic reports that display multiple data points for each student: 
Time Online, Pages Read, Notes Written, Number of Logins, Revisits. Each score is colour-coded as 
follows: 

• Green: A score two or more standard deviations above the class mean; 

• Light green: A score between one and two standard deviations above the class mean; 

• Black: A score within one standard deviation of the class mean; 

• Light red: A score between one and two standard deviations below the class mean; 

• Red: A score two or more standard deviations below the class mean. 

Teachers who review these data can quickly scan for students who have multiple red and light red 
scores. The red text helps identify the individuals most likely in need of assistance or coaching. In this 
fashion, the existence of a single low score is not necessarily cause for concern, but a string of low 
scores provides an impetus for a deeper analysis of student work. 

3. Exploration of online environment designs that are conducive to revisiting. 

Given that distributed activity is widely considered to be educationally beneficial, an argument can be 
made that the process of revisiting deserves deeper consideration. How can we design the user 
interface of a learning environment to facilitate the revisiting of important content? How much effort 
(perhaps measured in mouse clicks) does it take to retrieve a previously accessed page? How can we 
minimize the effort and time required to access earlier material? Often, web-based learning 
environments unintentionally discourage revisiting. For example, sometimes students cannot view 
previously examined material without leaving the current page. This is not only inconvenient, but also 
imposes additional cognitive load; revisiting requires the learner to navigate away from the current 
page, find the old content, review it, and then navigate back to the original location. A similar problem 
can surface when learners want to revisit content while in the midst of editing text. Many environments 
require the student to first save (or abandon) their text and exit the editing space before they can access 
the desired material. Again, this is inconvenient for the learner and a significant impediment to 
revisiting. One could imagine running experiments that use revisiting analytics to compare the incidence 
of student revisiting in differently designed environments, with different software supports. In this case, 
the purpose of the analytic tools would not be to conduct a diagnostic assessment of the learner, but 
rather to evaluate the effectiveness of different software designs, with the goal of making revisiting as 
effortless and transparent as possible. To be clear, this is not a call for a wholesale return to traditional 
experimental approaches, but rather a suggestion that the field of learning analytics invent new ways to 
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infuse experimentation into their studies. For example, in our studies of Pepper online learning courses, 
we have developed the idea of a "pseudo-double-blind" study. In a study of this sort, half the students in 
an online class are randomly assigned "Interface Feature A," and the other half (the control) are 
assigned "Interface Feature B." Neither the students nor the instructor are aware of the subtle 
differences in the software. Both groups are exposed to the same teacher and same content. Post-hoc 
comparisons of activity patterns across many such courses then allow us to draw causal inferences 
regarding the effectiveness of different interface elements. 

To conclude, Svihla, Wester, and Linn have conducted an interesting study that illustrates how learning 
analytics, grounded in an understanding of learning theory, can provide useful information about 
student activity. The study also highlights some of the challenges that learning analytics researchers 
face. Much work still needs to be done, but the authors have produced research that can serve as a 
foundation for follow-up investigations. By inventing a revisiting analytic tailored to the content of a 
particular unit, they have set the stage for better diagnostic tools and learning environments that more 
effectively embody the principles of distributed practice. 
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